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Preface 
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Foreword 

Why Evaluate? 



Program evaluation is one of the most powerful tools available to a project 
manager. It can be used to assess whether a project is being implemented as 
planned, and to assess the extent to which stated goals and objectives are 
being achieved. It allows project staff to answer the question, "Are we doing 
for our students what we said we would?' 

Evaluation data, therefore, can provide the basis for planning and decision- 
making relative to program improvement. This manual has been prepared to 
provide project directors and evaluators of vocational and adult education 
programs with suggestions and guidance for planning and carrying out such 
an evaluation. 
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I. Introduction 



A. Purposes of This Evaluation Guide 

The goal of this Guide is to help local adult and vocational education programs Improve their 
approach to evaluation. At the local program level, there is a need to obtain more 
comprehensive data in order to assess effectiveness and improve planning and management. 
The Guide presents an overall framework which can be used by program staff and evaluators 
to help establish the scope of their evaluations and to give direction to meeting local 
evaluation needs. As resources become more scare, programs will be under pressure to 
document their effectiveness In order to maintain continued funding and attract new funding 
sources. The framework presented In this guide can be used to provide this much needed 
documentation. 

What the Guide Is Intended to Do 

In the broadest sense, the Guide will provide guidance and assist local program staff and 
evaluators to design and carry out their local evaluations. Thus, the Guide Is Intended to: 

» provide direction for planning a relevant evaluation which meets the unique 
needs of each local program; 

» provide a compendium of evaluation questions around which a local evaluation 

can be structured; 

► provide sample Instruments for collecting data on program participants and for 

documenting program process and procedures; and 

provide specific guidance for carrying out collection activities; such bs 
follow-up data from former students or trainees, and from employers. 



What the Guide Is Not Intended to Accomplish 

The Guide Is not Intended to be a "textbook" about evaluation In general, nor about 
evaluation of vocational and adult education programs In particular. 

Specifically, it Is not intended to be: 

► a "cookbook" for evaluating local programs with specific recipes and menus for 
each aspect of an evaluation; 

» considered a lock-step methodology with a compendium of precise Instruments 

for exa mining every aspect of a specific program; nor 

> a limiting or restrictive directive on what should or should not be evaluated. 
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Since the features and needs of projects vary, specific measures and procedures for assessing 
vocational and literacy skills are not goals of this Guide. Program staff should f«iS 
adapt the materials In this Guide to their own needs Sh0Uld * 



B. Audiences 

The primary audience for this Guide Includes: 
► local program directors; 

*■ program staff; and 

*■ evaluators. 



Others who may profit from using the Guide are state program administrators and 
coordinators who wish to know more about the Impact of local programs across their States 
and can use the Guide to help plan evaluations of state-wide pro^s 

In summary, this document is called a Guide for deliberate reasons. Local program staff and 
oSSnm fram ? ro, : k ’ valuation questions, sample tastnLe^folfol-up 

Sfth?T, and 0ther a ^? ccts of data coUecti °n to address the issues and match the needs 
P*' ogram ; The evaluation questions and the sample Instruments presented In 

to T n0t tat ;? ded ? * tocluslVe * me ans. They provide a £££$& 

to facilitate the process of formulating evaluation questions and focusing the evaluation. 



• • * 



^ 1 ! n . CXt f C ^ a ? t ® r ° fthls G . uide Presents a six-step evaluation process. This process is general 
111 a«u arscusses now an evaluation should be planned and conducted Chapter ill 

on the other hand, sets forth a framework for specifically evaluating loc* £gS 

suggestions and guidance for what aspects oftoe pro^raS^ouldte 

ted The U,ha ^ qUC !SS S Sh0Uld h* ad£ toessed, and whatdSL should be 

collected The general process described In Chapter II should be applied to the specific 

iv re S, C 'r P , t rt r “ t "° rd ' r “ “ d °»>*><* « comprehensive evah£S£. 

Chapters IV - Vll of the Oulde provide guidance In planning for each component of the 

^““preSSS' 01 ' ChaPter VHI dlacu ““ *>ow the data and the evalJSon 
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n. Evaluation Process and Plans 



Program evaluation Is a basic tool for program management and Improvement. Central 
questions which an evaluation can help answer Include: 



► How Is a program being Implemented? (What is taking place?) 

► To what extent have program objectives been met? 

► How has a program Impacted on Its target population? 

► What contributed to successes and failures? 



» What changes and improvements should be made? 

Evaluation involves the systematic and objective collection, analysis, and reporting of 
information or data. Using the data for program Improvement and increased effectiveness 
then Involves Interpretation and judgement based on prior experience. 



A. Overview of the Evaluation Process 



The evaluation process can be described as involving six progressive steps. These steps are 
shown In Exhibit 1, and are discussed below. It Is Important to remember that Initiating an 
evaluation cannot wait until a program Is nearing completion. An evaluation should be 
Incorporated Into overall program planning, and should be initiated when the program begins 
operation. In this manner, program processes and activities can be documented from their 
beginning, and baseline data on program participants can be collected as they enter the 
program. 

Step 1: Defining the Purpose and Scope of the Evaluation 

The first step In planning an evaluation is to define its purpose and scope. This helps set the 
limits of the evaluation, confining it to a manageable size. Defining its purpose Includes 
deciding on the goals and objectives for the evaluation, and on the audience for the 
evaluation results. The evaluation goals and objectives may vary depending on whether the 
program being evaluated is new and is going through a tiy-out period for which the planning 
and im plementation process needs to be documented, or If a program has been thoroughly 
tested and needs documentation of its success before information about the program is 
widely disseminated and adoption by others encouraged. 

Depending on the purpose, the audience for evaluation may be restricted to the project 
director and his/her staff, or may Include a wider range of individuals, from agency 
administrators and decision-makers, to planners and other officials at the local, state, or 
national level. 

The scope of the evaluation depends on the evaluation’s purpose and the Information needs 
of its intended audience. These needs determine the specific components of a program which 
should be evaluated and on the specific project objectives which are to be addressed. If an 
evaluation of a program has recently been conducted, a limited evaluation may be designed 
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EXHIBIT! 

Steps In Ths Evaluation Procass 
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to target certain parts of the program which have been changed, revised, or modified. 
Similarly, the evaluation may be designed to focus on certain objectives which were shown 
to be previously only partially achieved. Thus, this step will define exactly which aspects of 
the program are to be evaluated. Costs and resources available to conduct the evaluation 
must also be considered in this decision. 

Step 2: Specifying the Evaluation Questions 

Evaluation questions grow out of the purpose and scope specified in the previous step. They 
help further define the limits of the evaluation. The evaluation questions will be answered 
in the evaluation report and should be formulated to address the needs of the specific 
audience to whom the evaluation is directed. Evaluation questions should be developed for 
each component of the program which falls into the scope which was defined in the previous 
step. For example, questions may be formulated which concern the adequacy of the 
curriculum and the experience of the instructional staff; other questions may concern the 
appropriateness of the skills being taught in relation to employment opportunities in the local 
community; additional questions may relate to the appropriateness of the individuals being 
recruited for the program with respect to literacy and employment status; and finally, 
evaluation questions may relate to the extent to which participants are achieving the goals 
of the program, such as obtaining a GED, high school diploma, or specific job skills, stable 
employment and increased earning capability. 

A good way to begin formulating evaluation questions is to carefully examine the program’s 
objectives; another source of questions is to anticipate problem areas concerning program 
Implementation. Importantly, the audience for or expected users of the evaluation should be 
invol/ed in developing the evaluation questions. This should never be left solely to the 
outside evaluator, no matter how familiar he or she is with the program. Once the evaluation 
questions are developed,. they should be prioritized and examined in relation to the time and 
resources available. Once this is accomplished, the final set of evaluation questions can be 
selected. 

Step 3: Developing the Evaluation Design and Data Collection Plan 

This step involves specifying the approach to answering the evaluation questions, including 
how the required data will be collected. This will involve: 

► specifying the data sources for each evaluation question; 

► specifying the types of data, data collection approaches, and instruments 
needed; 

► specifying the specific time periods for collecting the data; 

» specifying the staff members who will be assigned to collecting the data, and 

how their data collection responsibilities relate to their other project 
responsibilities; and 

► specifying the resources which will be required to cany out the evaluation. 

The design and data collection plan is actually a roadmap for carrying out the evaluation. 

An Important part of the design is the development or selection of the Instruments for 
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collecting and recording the data needed to answer the evaluation questions. Data collection 
instruments may Include record-keeping forms, questionnaires. Interview guides and 
assessment measures of literacy and vocational skills. Some of the Instrumentation may 
already be available, i.e., forms used for recordkeeping and management purposes, such as 
recruitment or intake forms, etc. Some of these forms will have to be modified to meet the 
evaluation needs. In other cases, new Instruments will have to be created. 



In designing the Instruments, the relevance of the items to the evaluation questions and the 
ease or difficulty of obtaining the desired data should be considered. Thus, the Instruments 
should be reviewed by the project director and staff members to ensure that the data can be 

o talned in a cost-effective manner and without causing major disruptions, or Inconveniences 
to the project. 



Step 4 ; Collecting the Data 

Data collection should follow the plans developed In the previous step. The individuals 
assigned to the various data collection tasks need to be thoroughly trained In the data 
collection requirements and procedures. Only by following standardized procedures will the 
data be reliable and valid. Following training, the project director needs to monitor the staff 
to ensure that they are accomplishing their data collection assignments according to the 
specified time schedule. The data should be recorded carefully and neatly so they can be 
read and Interpreted during the analysis stage. Proper record-keeping and filing are similarly 

S ?u ***5 ^ dat ^ not lost or mls P laced - Problems should be Immediately 
brought to the attention of the project director and evaluator. Deviations from the date 

collection plan should be documented so that they can be considered In analyzing and 
interpreting the data. & 



Step 5; Analyzing the Data and Preparing a Report . 

This step involves tabulating, summarizing, and interpreting the collected data in such a way 
as to answer the evaluation questions. These procedures should be compatible with the type 
and amount of data which were collected, and the goals and objectives of the evaluation. 
Appropriate descriptive measures (frequency and percentage distributions, central tendency 
and variability, correlation, etc.) and inferential techniques (significance of difference between 
means and other statistics, analysis of variance, chi-square, etc.) should be used to analyze 
the data. The local evaluator should have responsibility for this aspect of the evaluation. 

The evaluation will not be completed until a report has been written and the results 
communicated to the project director and other appropriate administrators and 
decision-makers. In preparing the report, the writers should be clear about the audience for 
whom the report is being prepared. Two broad questions need to be considered: (1) What 
does the audience need to know about the evaluation results? and (2) How can these results 
be best presented? Different audiences need different levels of information. Administrators 
need general information for policy decision-making, while program staff may need more 
detailed information which focuses on program activities and effects on participants. 

The report should cover the following: 

► The goals of the evaluation; 

► The procedures or methods used; 

► The findings; and 

► The implication of the findings, including recommendations for changes or 
improvements in the program. 

importantly, the report should be organized so that it clearly addresses all of the evaluation 
questions specified in Step 2. 

Step S: Using the Evaluation Report for Program Improvement 

The evaluation should not be considered successful until its results are used by program 
managers and decision-makers for program Improvement. After all, this is the ultimate 
reason for conducting the evaluation. The evaluation may Indicate that a program activity 
is not being Implemented according to plan, or it may Indicate that a particular objective is 
not being met. If so, it is then the responsibility of the project director to make appropriate 
changes to remedy the situation. Project directors should never be satisfied with their 
programs. Improvements can always be made, and evaluation is an important tool for 
accomplishing this purpose. 

B. Planning the Evaluation 

The evaluation should be conducted by an independent, experienced evaluator. This 
individual will provide the expertise for an evaluation which is comprehensive, objective, and 
technically sound. The project director and her/his staff must work closely with the 
evaluator beginning with the planning stage to ensure the evaluation meets the exact needs 
of the program. 



Adequate time and thought for planning an evaluation Is essential, and will give the prelect 
director and staffan opportunity to develop Ideas about what they would like the evaluation 
o accomplish. The evaluation should address the goals specified in the project application 
and management plan. In some projects, however, one or more goals or objectives may 
require special attention. Some activities or Instructional strategies may have been recently 

Fo? Sta f ** a ^® re ofsome special problems which should be addressed 3 ^ 

example, there might have been a recent breakdown In co mm unication between 
rnsnuctom; or the characteristics of students In recent Instructional groins r^tW 
anS^rhlfS the past, having implications for vocatkL^ra^ orS 

whM^the are examples of things which should be^oSidered 

OT h h^if Sm! 2,^ t° r selects an evaluator. The evaluator must then familiarize himself 
or herself with the special Issues of concern on which the evaluation should focus. 

Thus, the initial step of the evaluation process involves think ing about anv soecial needs 
»?P * Panning the overall evaluation and 

identified and evaluation questions which focus on curriculum and instructional materials 
might suggest that an evaluator is needed with particular expertise In those areas. Similarly 

mSmdSS t s n h ed tif n ev f uatlon ^ Wch squires frequent on-site observation of teaching 
methodologies by the evaluator, then this will help the project director focus on hWnf 
someone located nearby so that travel costs can be kept to a minimum ® 

ind'^w dCfl ? n ? SC ° P ' involves ^ts. identifying specific areas of Inquiry 

and deciding on what parts of the program and on which objectives the evaluation will focuZ 

d nf S n0t anSWCr q uestlon of how the evaluation wifi be conducted In 
estabhshlng the scope, one is actually determining which components or parts of the program 
will be evaluated, and Implies that the evaluation may not cover every aspect and actfrtty^ 

C. Selecting the E wlintor 



Selecting an evalu *or Is one of the most Important elements in ensuring a technically sound 

and useful evaluate '.. The basic criteria suggested for selection are as follows: 



Experience in conducting evaluations of similar types of vocational and adult 
education programs; 



► Skills in evaluation design, constructing data collection Instruments, collecting 
data, managing and maintaining quality control over data collection, analyzing 
data, and writing reports; 



* Ability and willingness to work directly with the project director In order to 
design the evaluation, oversee Its Implementation, and prepare a report; 

Knowledge of the educational needs of special populations. If applicable; and 

* Experience in collecting data from employers and community and business 
groups, if appropriate. 



The evaluator must be available for the complete time period required for the evaluation at 
f ? , at flts * ,a8lc b ud 8 et resources and the number of evaluator-days projected In the 
initial planning. The evaluator should also reside within reasonable distance of the prelect 
so that travel costs are minimal and scheduling work sessions is not a problem. 
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Once an individual has been selected and has agreed to become the local evaluator for the 
project, a contract and work plan must be developed so that expectations, roles, and 
responsibilities are clear to all parties. 

D. Specifying the Roles of Project Director, Staff and Evaluator 

In order for the evaluation to be planned and carried out effectively, the roles of the project 
director, staff, and evaluator must be made clear to all parties. The evaluator should be 
responsible for specifying the design and developing the data collection Instruments, training 
project staff to collect the required data, analyzing the data, and preparing a comprehensive 
report. The project director should work with the evaluator in specifying the objectives and 
scope of the evaluation, and assigning and supervising the project staff in carrying out their 
assigned data collection and record-keeping tasks. The project staff should be assigned the 
major data collection and record-keeping tasks for the evaluation. This will increase their job 
responsibilities, but is the most cost-effective way of collecting the required data. More 
specifically, the suggested roles of the project director, the evaluator, and the project staff are 
listed below: 

The project director will: 

► recruit and hire an experienced evaluator; 

► work with the evaluator to define the objectives and scope of the evaluation; 

► work with the evaluator to define the evaluation questions which will be 
addressed; 

, review and approve the proposed data collection instruments and procedures 
to ensure that they are compatible with project activities; 

► assign and supervise project staff for data collection and recordkeeping; 

► approve the schedule and expenditures for the evaluation; and 

► approve the final evaluation report. 



The evaluator will: 



equation- **** pr0ject director t0 define the purpose and scope of the 

ad^2S <the Pr0 ^ CCt director t0 specify the evaluation questions which will be 

► develop the evaluation design: 

select and/or develop the data collection instruments and procedures 
including the schedule and calendar for all data collection: 

► train the project staff in the procedures for collecting the required data: 

* analyze the collected data to answer the evaluation questions: and 

* prepare the evaluation report. 

The project staff will: 

► collect the evaluation data; 

' S^ , tud«“ t aS Cklng r ' COrdS “ “ COnduCt ^ of 

► assist the evaluator in tabulating and summarizing the collected data. 

E. Allocating Resources 

Critical questions in planning the evaluation concern how much staff time and financial 

°« th ' era “ and what rcsoure ' s « — dSoSsj 

carry « out. These two questions need to be addressed by the protect director and the 

car }^ 111 desl S n Process. Many evaluators wlU raise these questions in the initial 
exploratory discussions concerning the evaluation. Preliminary planning will have to be 

^T pU r She ^ bef ® re a 111131 decision can be made on both the role mJdkvel of effort of tte 
evaluator and each project staff member. 

A variety of design decisions must be made during the planning stages that affect the 
aUocation of both financial and staff resources. Each decfsZ^^e sM/Sa^ 
roles and functions, level of effort, and resources, and ultimately determines the oveSuscope 

ac a a record-keeping procedure must be developed to^ainta^ 

accurate equation data. A tracking system is also needed to manage follow-up surveys of 
°™ er students. These record-keeping and tracking systems, as well as the actual data 

taVo1 - staff f d material costs. These costs need to bfcon^ S 
denning the final scope of the evaluation. 
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♦ * * 



To summarize, this chapter presented a general process which focused on how ah evaluation 
may be planned and carried out. The next chapter, and In fact the remainder of this Guide, 
discusses what program components, evaluation questions, and data should be Included in 
an evaluation of a local vocational or adult education program. 
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[. Evaluation Framework 



This chapter presents a framework for evaluating local programs which combines outcome 
evaluation with process evaluation. An outcome evaluation attempts to determine the extent 
to which a project’s specific objectives have been achieved. On the other hand, the process 
evaluation seeks to describe the program which was implemented, and through this, attempt 
to gain an understanding of why the objectives were or were not achieved. 

Evaluators have been criticized in the past for focusing on outcome evaluation and excluding 
the process side, or focusing on process evaluation without examining outcomes. The 
framework presented here incorporates both the process and outcome side. In this manner, 
one can determine the effect (or outcome) of a program, and also understand how the 
program produced that effect and how the program might be modified to produce that effect 
more completely and efficiently. 

In order to focus on both program process and outcomes, an evaluation should be designed 
in which evaluation questions, and data collection and analysis, address the following: 

► Program Environment; 

► Program Participants; 

► Program Implementation; and 

► Program Outcomes. 

These components may be thought of as being linked in the following manner. 




H 



Program Imptomsntstton H Program OutcomM 




Using this framework, descriptions are prepared of the environment the participants, and 
the program activities and services which are implemented. Outcomes of the program are 
ainn aosessed. The description of the environment participants, and activities and services 
are used to explain how the outcomes were achieved and to suggest changes which may 
produce these outcomes more effectively and efficiently. 
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Each evaluation component is described below. 

Environment 



^component defines toe environment In which a program is Implemented. This Includes 
the community In which the program is located, the economy of the community, the lob 

tarflS* f "* 1 Hf educatlonal andjob training needs of the program’s 
Population. Understanding the environment in which the project is locatedis 
Important to assessing and Interpreting the outcomes of the program, especially such 
outcomes as Job placement, Job retention, or Job advancement. Data concerning this 
component are collected through a needs assessment which should be completedprior to 
applying for funding. These data are generally presented In a grant application in order to 
make the case for the need for the program In the first place. 

Participants 

P* C< S nCnt dCfln ” characteristics of the participants, including literacy, educational 

ievel, previous vocational training, and work history. Data on the characteristics of 

** as of Program recruitment and intake activities. In 

addition to their use for descriptive purposes, these data are useful for comparisons with 
similar data collected at program completion and at follow-up data collection periods. 

Program Implementation 

how «“ activities of the program are Implemented. Including 
recruitment, intake and assessment; Instruction; counseling and other support services- and 

SSr 1 and - i0h Placcment ’ applicable. In this manner, the oSt^m^r rosSs 
achieved by the program can be attributed to what actually has taken place, rather than what 

° CCUr ‘ component 3130 addresses the questions of what services and 
top^ment°(d Uie pr0gram have been ^ ta plemented. partially Implemented, and not 

Outcomes 

ms component concerns the effects that the program has on its participants, and to what 
b 0 ® 1 me J, stated objectives for program participants. At program 

completion, data may be collected on such variables as on rates of program completion, 
achievement of Instructional objectives and competencies, vocational sWs and knowledge 
Enghsh language proficiency, placement In a Job. continued education/lnstruction.lnd 
satisfaction with the instruction received. 

If applicable, follow-up data may be collected at appropriate points In time following program 
completion on employment rates of former participants, wages per hour, numbe? of hours 
worked per week, number recelvlngjob promotions, number receiving wage Increases, ratings 
of Job satisfaction, and ratings of employer satisfaction with vocational and literacy skills. 

Using the above four evaluation components, a comprehensive assessment of a program may 
be designed. Not only will this evaluation approach allow the project staff to determine the 
extent to which project goals and objectives are met. but will also enable them to understand 
tath^ftf C ° UtCOmCS Were achleved 311(1 10 changes for program Improvement purposes 
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* * * 



The evaluation framework presented above should be Implemented using the six-step process 
/A described In Chapter II. The framework describes what should be Included In the evaluation; 
the six-step process describes how the evaluation Is to be planned and carried out. 
Guidelines for defining the scope of the evaluation, specifying evaluation questions, and 
developing the data collection plans for each of the four evaluation components are discussed 
in the following chapters. 
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IV. Program Environment 



This chapter focuses on the collection of descriptive Information concerning the educational, 
economic, and employment environment of the community In which the program Is located. 
These data provide a context against which program outcomes may be measured. 

Information on euacational, economic, and employment characteristics of the community will 
most often be collected as part of a needs assessment which is generally conducted prior to 
planning a project. In fact, the needs assessment results are often Included as a part of a 
program’s grant proposal or application for funding. Thus, the needs assessment data may 
satisfy the requirements of this component of the evaluation. 

In focusing on the desired Information for this aspect of the evaluation, the project director 
and evaluator should decide on the purposes that Information on the environment of the 
program may play In the overall evaluation plan. This will define the scope of this part of the 
evaluation. From this, the relevant evaluation questions may be developed. 

An Illustrative set of questions relating to the program environment is provided In Exhibit 2. 
The final set of questions should be defined and developed by the project director and 
evaluator so that they meet the needs of their local program. 

After the questions arc specified, the evaluator should Identify the specific variables which 
are applicable to the questions, and the possible sources of data which may be used to 
answer the questions. Examples of variables and data sources are also shown In Exhibit 2. 
Following this, a strategy for data collection arid the data collection Instruments themselves 
need to be developed. Relevant data may be available from such agencies as the local school 
system, the loczil economic development agency, local or state employment department, and 
Chamber of Commerce. 

The findings which result from this part of the evaluation will eventually help explain the 
results of the outcome evaluation. For example. If project goals concerning Job placement are 
not achieved, it may be because the program is providing vocational training for occupations 
which are inappropriate for the present economic environment of the local community (no 
Jobs are available). Thus, the evaluation results can help the project director make 
progra mmin g decisions to rectify the situation. 
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1 . 



2 . 



3 . 



4 . 

5 . 



6 . 



EXHIBITS 

Program Environment: Examples of Evaluation 
Questions, Variables, and Data Sources 



Education Questions 



Variables 



What are the general 
characteristics of the community 
and target population served by 
the program? 



What are the employment and 
economic characteristics of the 
community In which the 
program Is located? 

How many members of the 
target population In the 
community need employment 
and related services? 



How many adults need literacy 
and/or vocational training? 

What kinds of jobs are available 
In the community for which the 
program could train adults? 



Total population; 
number of Individuals 
In target population, 
demographics of the 
target population. 

Unemployment rate; 
mean family Income of 
general population and 
target population. 

Number of adults 
seeking employment; 
number of adults 
unemployed; number of 
adults underemployed. 

Number of adults 
needing training. 

Jobs available In 
community. 



Possible Data 
Sources 

Published Census 
Reports 



Local Labor 
Department 



Local Labor 
Department 



Local Labor 
Department 

Local Labor 
Department 



Which areas of vocational Types of jobs available 

training are most needed In the In community 
community 



Local Labor 
Department 
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V. Program Participants 



This hapter concerns that part of the evaluation related to the number and characteristics 
of the program’s students or trainees. These data will be collected during recruitment periods 
or upon entiy to the program. Most student data will be descriptive In nature, such as age. 
ethnicity, and prior education or occupational training. Some data, however, will be baseline 
measures of variables related to program objectives. These data will be compared to data 
collected at completion of the program to determine effects. Examples of these data are 
literacy levels, vocational skills, employment, status, and job wages. 

The specific student data to be collected should be determined by the project director and 
evaluator, and depends on the issues which they choose to address. From this, a set of 
evaluation questions may be developed by the evaluator which focus on these issues. This 
then leads to specification of variables, and the development of the data collection plans and 
instruments. 

Evaluation questions concerning students or trainees are shown In Exhibit 3, along with 
examples of the relevant variables and data sources. A particular local program may wish 
to add, modify, or delete questions from this listing. 

A program intake form can generally be used to suppfy most. If not all, of the Information 
needs concerning student characteristics. The evaluator should examine existing form(s) 
used by the program to determine whether all data needs are satisfied. If not, these forms 
should be modified so that the additional required information is collected. 

A sample Intake form is shown in Appendix Al. Additions, deletions, and modifications to 
the form may be made as desirable. 

Each entering student may be asked to complete a Student Intake Form during the 
recruitment or intake process, or during the first class session. Alternatively, staff members 
may interview entering students and record the data on the Intake Form themselves during 
the intake interview. The evaluator or a designated staff member will then be responsible for 
summarizing the data across students. 
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I****TT 3 

Program Participants: Examples of Evaluation 
Questions, Variables and Data Sources 



Evaluation Questions 

1 . How many individuals are 

i enrolled? 

2. What are the demographic 
• characteristics of the 

participants? 



3. What is the level of English 
language proficiency of 
participants before they enter 
the program? (See Appendix 
B.) 

4. What is the level of native 
language proficiency of limited 
English proficient (LEP) 
students before they enter the 
program? 

5. What are students’ reasons 
for enrolling In the program? 

6. What are the vocational skills 
of participants at program 
entry? 

7. How much previous 
vocational training do 
participants have at progr am 
entry? 

8. What are the job interests and 
goals of participants at 
program entry? 

9. What are the employment 
status and wages of 
participants at program 
entry? 



Variables 

Number of individuals 
enrolled 

Age, Sex, Racial/Ethnic 
Group, Years of Formal 
Schooling, Achievement of 
High School Diploma or 
GED Certificate, Family 
Status. Number of 
Dependents. 

Scores on measures of 
English Language 
Proficiency 



Scores of measures of 
Native Language 
Proficiency 



Students’ goals/objectives 



Vocational Skills 



Years of Vocational 
Training; Areas of 
Vocational Training 



Job Interests; 
Job Goals 



Employment Status; Public 
Assistance and 
Unemployment Benefits 
Received: Hourly Wages 
Prior to Program Entry. 
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Possible Data Sources 

Program Records 

Student Intake Form 



English Language 
Proficiency Test Given to 
At Intake 



Native Language 
Proficiency Test Given To 
LEP Students At Intake 



Student Intake Form 



Student Intake Form, or 
Pretest of Vocational 
Skills 

Student Intake Form 



Student Intake Form 



Student In Jce Form 
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VI. Program Implementation 



This chapter focuses on documenting how a vocational or adult education program is 
implemented and the services which are provided. This includes recruitment, intake and 
assessment activities; literacy, basic skills, and vocational instruction; job counseling and 
other support services; Job development; and job placement. The data to be collected will 
focus on what has actually taken place, rather than what was originally planned. With 
proper documentation, the project staff can assess the changes needed for future program 
improvement. 

The project director and the evaluator should work together to decide how the specific 
activities and services of the program are to be described or documented, and the level of 
detail desired. Once this has been determined, the project director and evaluator should 
develop a set of evaluation questions which focus the data collection requirements. Sample 
questions, shown in Exhibit 4, cover a wide range of program activities. The project director 
and local evaluator should review these and select the ones which are relevant to their 
program. In addition, modifications and additions to th^se questions should be developed 
to fit the needs of the individual program. 

The evaluator will be responsible for developing the plans and Instruments for collecting the 
necessary data. Appendix A2 contains an example of one such instrument, a sample 
Program Profile whose purpose is to collect and o mpile the data needed to describe the 
program’s processes and services. Appendix A3 contains a sample Student Instructional 
Status and Attendance Form which can be used to collect data on instructional services 
received by individual students. These sample forms can be adapted ro fit the needs of the 
specific evaluation questions which have been developed by the project director arid 
evaluator. 

Data for the Program Profile can be collected from project records or directly from the project 
director and his/her staff. Two strategies are available. The evaluator can review project 
records and interviev r the project director and staff to complete the form, or the project 
director and staff members can respond to the form in writing. The latter data collection 
strategy is probably more cost effective since it minimizes the time of the evaluator. 
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EXHIBIT 4 

Program Implementation : Examples Of Evaluation 
Questions, Variables, and Data Sources 



Evaluation Questions 


Variables 


Possible Data Sources 


1 . What are the objectives of the 
program? Are these objectives 
clearly stated and measurable? 


Program Objectives 


Project Application; 
Management Plan 


2. In what adult education/ 
vocational training areas does 
instruction take place 


Instructional Area 


Project Application and 
Records 


3. How long is each training cycle? 


Number of Weeks or Months 


Project Schedule 


4. What is the total number of hours 
of instruction provided to each 
student? 


Number of Training Hours 


Project Schedule 


5. What are the instructor/student 
ratios? 


Number of Students; 
Number of Instructors 


Project Records 


6. To what extent is enrollment 
meeting the program goals? 


Program Objectives; 
Number of Students 


Project Application; 
Project Records 


7 . How is recruitment conducted? 
What criteria are used for 
accepting individuals? 

Are criteria appropriate? 


Recruitment Procedures; 
Enrollment Criteria 


Project Application; 
Observation 


8. How is the project staffed? 

Is staffing appropriate and 
sufficient for project operation? 


Number of Staff by Position 


Project Records 


9. What are the qualifications of the 
staff? Do staff have necessary 
qualifications to meet the needs of 
the students? 


Background and Experience of 
Staff 


Employment Applications; 
Staff Resumes and 
Interviews 


10. What kind of staff development 
and training are provided to staff 
members? Are development and 
training appropriate and sufficient? 


Staff Development and 
Training Activities 


Project Records; Staff 
Interviews; Observations 


1 1. What specific vocational 

competencies are addressed by 
the training program? 


Vocations Competencies 


Project Application; Project 
Records 


12. What specific basic skill 

competencies are addressed by 
the program? Are competencies 
appropriate and sufficient? 


Basic Skills Competencies 


Project Application; Project 
Records 


13. What criteria are specified for 
successfully completing the 
program? Are criteria 


Completion Criteria 


Project Application; Project 
Records 



appropriate? 



26 



Evaluation Questions 

14. Docs the training curriculum as 
implemented follow the project 
plan? Is curriculum appropriate? 

15. What training methods and 
materials are used? Are the 
methods appropriate? 

16. What job counseling services are 
provided to students? 

Are these services appropriate and 
sufficient? 



17. What job development activities 
are implemented by the staff? 

Are these activities appropriate 
and sufficient? 

18. What job placement services are 
provided to students? Are these 
services appropriate and 
sufficient? 

19. How are employers and the 
general community involved in the 
training program? Is this 
involvement appropriate and 
sufficient? 

20. What facilities and equipment are 
used by the program? 

Are the faciliUes and equipment 
appropriate and sufficient? 

21. What is cost of training per 
student? 



Variables 

Description of Training 



Description of Training; 
Methods and Materials 



Description of Job Counseling 
Services; Number of 
Counseling Sessions; Number 
and Percent of Students 
Counseled; Number of 
Counseling Hours Provided 

Description of job 
Development Activities; 
Number of Employers 
Contacted 

Description of Job Placement 
Services; Number and Percent 
of Students Served 



Number of Group and 
Individual Meeting Held; 
Number of Phone and Written 
Communications 



DescripUon of Facilities and 
Equipment 



Total Project Funding; 
Number of Students 



Possible Data Sources 

Project Application; Project 
Records; Staff Interviews 

Project Application; Staff 
Interviews; Observation 

Project Application; Project 
Records; Staff Interviews; 
Observation 



Project Application; Project 
Records; Staff Interviews; 
Observation 



Project Application; Project 
Records; Staff and 
Participant Interviews; 
Observation 

Project Application; Project 
Records; Staff and 
Employer Interviews 



Project Application; Staff 
Interviews; Observation 



Project Budget; Project 
Records 
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vn. Program Outcomes 



Program outcome data are used to determine the extent to which a program Is meeting Its 
goals and objectives for students or trainees. These goals and objectives may Include: 

► improved literacy and computational skills; 

► increased vocational skills; 

► stable employment; 

► Increased earning capability; and 

► greater career advancement opportunities. 

As with the other evaluation components, the project director and evaluator must workjolntly 
to define the scope of the outcome data to be collected. This should be accomplished by 
developing a set of evaluation questions to assess the extent to which the project’s goals and 
objectives are met. Examples of evaluation questions directed at the outcomes of programs 
are shown in Exhibit 5. Also shown are the relevant variables which relate to the questtons 
and possible sources from whom data may be collected. Project directors and local evaluators 
should carefully examine these questions and select those which are applicable to their 
program. Additional evaluation questions may also be specified which address any special 
Issues and concerns of the local program. 

Depending on the goals of the program, data required at program completion may include: 

► number of completers: 

*- achievement of basic skills objectives; 

► achievement of vocational objectives and competencies; 

► number of completers placed in Jobs related to vocational area in which 
trained; and 

► for those placed, wages per hour. 

Additional data will also be required depending on the final set of evaluation questions 
specified by the project director and evaluator. 

The data on achievement of educational and vocational training objectives and competencies 
should be available from project records. The measure(s) should be the same as were used 
at program entry. In this manner, pre/post-test differences can be calculated to measure 
change. Data on Job placements and wages may be obtained from progr am records or 
collected directly from the former trainees. 
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Data collected at follow-up may include: 

► number employed; 

► wages per hour; 

► number receiving wage Increase or higher paying Job since initial placement 
following training; and 

► number obtaining job promotions since initial placement following training. 

As with the program completion data, other data may be required depending on the 
evaluation questions which are specified. 

Follow-up data can be collected from former students or trainees or from their employers. 
First priority should be given to collecting data from former students, rather than from busy 
employers, It is suggested that contacts with employers to obtain project evaluation data be 
kept to a minimum. 

Data collection instruments which may be useful for evaluation purposes include a Student 
Follow-Up Telephone Survey, a Job Placement Information Form and Follow-Up 
Questionnaire, and an Employer Follow-Up Questionnaire. Samples of these data collection 
instruments are found in Appendices A4, A5, A6, and A7. These examples may be adapted 
to fit the needs of each local program. 

Appropriate data collection procedures and record-keeping systems need to be developed to 
ensure that program completion, placement, and follow-up data are collected, recorded, and 
filed. This will ensure that the data are readily available for tabulation and summary to meet 
the time schedule for the evaluation. Early planning by the evaluator and project director 
is one of the key elements to success in any data collection effort. This is especially true for 
the development of a record-keeping system for the collection of follow-up data. First, a 
tickler file or calendar must be established so that program staff can keep track of when each 
student or trainee has been placed in a job, and when follow-up is required. Without such 
a system, the staff will not be able to keep track of the appropriate time to follow-up each 
program completer. Second, an address file and a placement file for program completers 
need to be continually up-dated. Addressed and stamped post-cards may be given to 
students when they complete training so they can notify the program of address and job 
changes in a convenient manner. Alternatively, students can be asked to notify the project 
staff by telephone when they change addresses or jobs. Without these procedures, follow-up 
will be extremely difficult. 

Follow-up data may be collected via telephone interviews with former students. Before 
completing the program, each student should be told to expect these calls, and to keep the 
program informed of address and telephone number changes. On the other hand, collecting 
follow-up data from employers is a very difficult task, as employers are reluctant to give out 
personnel information, and generally are busy and do not like to be bothered. However, if 
collection of data from employers is required, program staff should lay the groundwork for 
follow-up by building a good rapport with employers when students are placed. By 
establishing good relationships at these times, follow-up data can be more easily collected. 
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The evaluator, with Input from the project director, should be responsible for developing the 
Instruments, procedures, and record-keeping systems. The project staff, under the 
supervision of the project director, should have the responsibility of collecting and recording 
the required data. Training the staff in all data collection and record-keeping procedures 
should be the responsibility of the evaluator. 

EXHIBITS 

Program Outcomes: Examples of Evaluation 
Questions, Variables, and Data Sources 



Evaluation Questions 

A. Program Completion 

1. How many students/ trainees 
completed the training program? 

2. To what extent were students' 
objectives met? What is degree of 
completers’ satisfaction with 
training program? 

3. To what extent did students 
increase their literacy skills? 



4. To what extent did 

students/trainees increase their 
vocational skills and knowledge? 



5. How many completers were placed 
in jobs following training? How 
many were placed in full-time and 
part-time training-related Jobs? 

6. What is the average gross wage per 
hour of completers who were 
placed in Jobs? 

7. What is the average number of 
hours worked per week of those 
placed in Jobs? 

8. How many completers were placed 
in or enrolled in further education 
or training programs following 
completion of training? 





Possible Data 


Variables 


Sources 


Number of Completions 


Program Records 


Ratings of Satisfaction 


Completers 


• 

Pre/Post Measures of 
Literacy Skills: 
Achievement of 
Training Objectives 


Program Records 


Achievement of 


Completers; 


Vocational Training 
Objectives and 
Competencies: Pre/Post 


Program Records 


Measures of Vocational 
Skills and Knowledge 




Number of Completers 


Completers; 


Placed In Jobs 


Program Reports 


Wages Per Hour 


Completers 


Number of Hours 
Worked 


Completers 


Number of Completers 
Placed or Enrolled for 
Further Education or 
Training. 


Completers 



Evaluation Questions 

B. Follow-Up 

1 . How many program completers 
were employed six months 
following Initial placement in 
training-related and non-training 
related, full-time and part-time 
jobs? 

2. What is average gross wage per 
hour of completers six months 
following initial placement? 

3. What is the average number of 
hours worked per week of 
completers six months following 
initial placement? 

4. How many completers are enrolled 
in school or other training program 
six months following Initial 
placement? 

5. How many completers are 
unemployed six months following 
initial placement? 

6. How many completers were 
employed at any time during the 
six month period following initial 
placement? 

7. How many completers obtained Job 
promotions during the six month 
period following initial placement? 

8. How many completers received 
wage increases or obtained a 
higher paying Job during the six 
month period following initial 
placement? 

9. How satisfied are former 
participants with the jobs six 
months following initial 
placement? 

10. How satisfied are employers with 
the vocational ''*'d literacy skills of 
the program completers they 
hired? 



Variables 


Possible Data 
8ourees 


Number of Completers 
Employed 


Completers; 

Employers 


Wager Per Hour 


Completers; 

Employers 


Number of Hours 
Worked 


Completers; 

Employers 


Number of Completers 
Enrolled in School or 
Training Program 


Completers 


Number of Completers 
Unemployed 


Completers 


Number of Completers 
Employed at Any Time 
During Follow-up 
Period 


Completers; 

Employers 


Number of Completers 
Who Obtained Job 
Promotions or Higher 
Level Jobs 


Completers; 

Employers 


Number of Completers 
Who Received Wage 
Increase 


Completers; 

Employers 


Ratings of Job 
Satisfaction 


Completers 


Ratings of Employee 
Satisfaction with 
Vocational Skills and 
English Language Skills 


Employers 



31 




Presentation of Findings 



Following data collection, the next steps In the evaluation process involve data analysis and 
preparation of a report. These steps require the expertise of an experienced evaluator who 
is objective and independent of the project and grantee. This is important for the 
acceptability of the report’s findings, conclusions, and recommendations. 

The evaluator will be responsible for developing and carrying out a data analysis plan which 
is compatible with the evaluation’s goals and audience. To a large extent, data will be 
descriptive in nature and may be presented in narrative and tabular format. However, 
comparisons of pre- and post-measures related to literacy and computational skills, 
vocational skills and knowledge. Job wages, etc., may require more sophisticated techniques. 
These depend on the nature of the data. 

The data will be analyzed to answer the evaluation questions specified in the evaluation plan. 
Thus, the analysis will allow the evaluator to: 

► describe the program environment; 

► describe the program participants; 

► describe the program activities and services; 

► describe the outcomes; 

» examine and assess the extent to which the program plan was followed; 

► examine and assess the extent to which the outcomes met the program goals 
and objectives; and 

► examine how the program environment, participants, activities, and services 
affected the extent to which the outcomes were achieved, and how the program 
can be Improved to achieve Increased success. 

An evaluation report will then: 

► describe the accomplishments of the program, identifying those elements of the 
program that were the most effective; 

► describe elements of the program that were Ineffective and problematic as well 
as areas that need modifications In the future; 

► describe the outcomes or the Impact of program services on the participants; 
and 

► document how the program environment, activities, and services contributed 
to the accomplishment or lack of accomplishment of the program goals and 
objectives. 
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The level of Information and documentation suggested in this Guidebook will make the report 
quite useful for making decisions about improving program services, instructional strategies, 
etc., for the future. In other words, the evaluation report is a decision-making and planning 
tool for the project director and his/her staff. The report will support the broad management 
functions of decision-making, program Improvement, accountability, and quality control. 

It is important to keep the report’s audience in mind. In most cases, this will be the project 
director and funding agency administrators. These individuals may not be experienced in the 
technical aspects of data analysis. Therefore, the report should present data in a 
straightforward manner, using tabular presentations to help the reader. Interpretations and 
conclusions drawn from the data and implications for future directions should be discussed 
in a clear narrative. An example of a report outline is provided in Exhibit 6 on the next page. 

In addition to the written report, the evaluator should provide an oral briefing to the project 
director, project staff, and appropriate administrators. This should take place after those 
individuals have had a chance to read the report. The briefing should first include a short 
presentation on the most Important findings, and on the conclusions and recommendations 
drawn by the evaluator. A question and answer period following that presentation will allow 
for discussion of those conclusions and recommendations that are most relevant to improving 
the program in order to yield greater benefits to participants, employers, and the 
community-at-large . The project director should then prepare and execute an action plan 
for implementing the appropriate recommendations. In sum, evaluation results should be 
used for program improvement. Learning from the past to improve future prog rammin g 
should be a primary management goal. 
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EXHIBIT 6 

Organization of Evaluation Report 

I. Executive Sununaiy (3-4 pages) 

► Purpose 

*■ Procedures 

*• Major Findings 

» Conclusions and Recommendations 

II. Introduction 

*■ Background of Program 

*• Objectives of the Evaluation 

- Summary of Procedures 

III. Description of Program Environment 

IV. Program Components 

► Descriptions of Program Activities and Services 

*• Deviations from Plan in Grant Application 

V. Number and Characteristics of Participants 

VI. Program Outcomes 

*• Program Completion 

*• Follow-Up 

VIII. Conclusions and Recommendations for Program Improvement 



34 



Appendices 



Appendix A: Sample Data Collection Forms 

Al. Student Intake Form A-l 

A2. Program Profile A-9 

A3. Student Instructional Status and Attendance Form A-23 

A4. Student Follow-Up Telephone Survey A-27 

A5. Initial Job Placement Information Form A-37 

A6. Job Placement Follow-Up Questionnaire A-41 

A7. Employer Follow-Up Questionnaire . A-47 

Appendix B: Measurement of English Language Proficiency of LEP Adults B-l 
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Appendix A1 



Student Intake Form 



Name: 



Last 
Address: 



House Number 



First 



Middle Initial 



Street 



City 

Telephone ( ) 

Date of Application: 



State 



Zip Code 
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A-l 



1 . 



Birthdate 



2. Sex 

Male 

Female 

3. Marital status 

Currently married 
Widowed 
Divorced 
Separated 
Never married 



4. Single Parent 
Yes 
No 



5. Race or ethnic group 

American Indian or Alaskan Native 
Asian or Pacific Islander 
Black, not of Hispanic origin 
Hispanic 

White, not of Hispanic origin 

6. Number of Years of School Completed 

None 

1-4 years 

5-8 years 

9 years 

10 years 

11 years 

12 years 

More them 12 years 

7. Highest Diploma/Degree Obtained 

None 

GED Certificate 

High School Diploma 

Technical Certificate 

AA Degree 

4 year College Graduate 

Other 
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A-3 



e. 



What is the most Important reason you enrolled in this program? 

Required by employer 

Required by another program or agency 

To satisfy family or friends 

To satisfy other personal or employment 
goals 

9. Do you speak a language other than English at home? 

Yes (continue below) 

No (skip to 10) 

What language do you speak? 

Spanish 

Aslan (Chinese, 

Vietnamese, Korean, etc.) 

Other 



How well do you read in that language? 

Veiy well 

Well 

Not well 

Not at all 



How well do you speak English? 

Veiy well 
Well 
Not Well 
Not at all 



10. Are you currently receiving any public assistance or public welfare payments, for 
example. Supplemental Security Income (SSI), Aid to F amilie s with Dependent 
Children (AFDC)? 

Yes (skip to 11) 

No (continue below) 

Did you receive in the last 12 months any public assistance or public 
welfare payments, for example. Supplemental Security Income (SSI), 
Aid for Families with Dependent Children (AFDC)? 

Yes 

No 
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1 1. Did you work full-time or part-time for pay last week? 



No (skip to 12) 

Yes (continue below) 

If employed, where: 

Describe job: 



Wages Per Hour: $ 

(Skip to 13) 

12. Were you looking for work during the last four weeks? 

Yes (continue below) 

No (skip to 13) 

Could you have taken ajob last week if one had been offered? 

No, already have ajob 

No, temporarily ill 

No, other reasons 

(in school, etc.) _____ 

Yes, could have taken Job 

13. How did you learn about this instructional program? 

Family members 

Friend/neighbor 

Work associate/employer 

Newspaper 

Radio 

Television 

Letter, notice or leaflet 
from school or other 

community organizations 

Other 
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A-5 



14. Previous Work Histoiy: 



15. Vocational Skills: 



16. Prior Adult Education/Vocational Training: 



17. Job Interests and Goals: 
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18 . 



a. 



b. 



c. 



d. 



e. 



f. 



6 - 



h. 



1 . 



J- 



k. 



1 . 



m. 



n. 



How important to you are the following reasons for enrolling in this program? 



Very 

Important 



Somewhat 

Important 



Not 

Important 



To improve 

reading/writing 

skills 

To improve math 
skills 

To improve speaking 
and listening skills 

To get a GED or high 
school diploma 

To prepare for a 
vocational or Job 
training program 

Togetajobora 
better Job or qualify 
for higher pay 

To improve Job 
performance 

To qualify for United 
States citizenship 

To feel better about 
myself 

To contribute better 
to my family and 
community 

To help my children 
with schoolwork 

To become less 
Independent on 
others for help 

To make others feel 
better about me 

To help continue my 
education 



Appendix A2 
Program Profile 



Program Name: 

City: State: 

Name and Title of Respondent: 

Area Code/Telephone No.: 

Date Form Completed: 



« 
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A-9 



1 . 



What is the maximum number of students/trainees your program can serve at one 
time? 

2. How many students/trainees are currently enrolled in your program? 

3. For Program Year 19 - 19 , specify the number of students /trainees served in 

your program in the following categories: 

a. Gender 

Male 

Female 

b. Age 

16 - 17 years old 

18 - 24 years old 

25-44 years old 

45 - 59 years old 

60 years and older 

c. Ethnic/Racial Group 

American Indian or Alaskan 

Native 

Asian or Pacific Islander 

Black, not of Hispanic origin 

Hispanic 

White, not of Hispanic origin 

d. Institutionalized 



Correctional 

Other 



4. 



Immediately Prior to Entering Program, 

a) Number of students who were employed 
full-time (35 or more hours per week) 

b) Number of students who were employed 
part-time (less than 35 hours per week) 

c) Number of students who were enrolled 
in school or other training program 

d) Number of students who were employed 

e) Number of students who were receiving 
public assistance benefits 

f) Number of students who were receiving 
unemployment Insurance benefits 



>. Summary of Scores on Program Outcome M easur es: 

Name of Number of Pre-test Mean Post- 

Type of Measure Instrument Students Mean Score test Score 

English Skills 

Math 

Vocational 



6. What is the average number of hours per week that students/trainees attend class 

or receive program services? hours 

7. Approximately how many of your current students/trainees receive instructional 
services: 

a. ) 5 or more days/week? 

2-4 days/week? 

1 day or less/week? 

b. ) During the day? 

In the evening? 
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8. To what extent are the needs of your students/trainees being met by services 
offered as part of your program? (For each row check the appropriate column.) 



Not at Somewhat Fully Don’t Know 

All 



a. ) Child Care 

b. ) Transportation = 

c. ) Health Services 

d. ) Counseling 

e. ) Job Search 

Assistance 

f. ) Financial Assistance 

9. To what extent do you rely on the following ways to recruit potential 

students/trainees? In responding, please assume that "very little" means that this 
approach accounts for less than 10% of your recruiting effort; that 'some means 
10% to 50% of your recruiting effort; and "a great deal" means over 50% of your 
recruiting effort. (Check the appropriate column for each row In the following list.) 

Not at Very Little Some A Great 
All Deal 

a. ) Announcements in mass media 

(TV, radio, etc.) 

b. ) Fliers, posters, mailings 

c. ) Referrals from welfare, 

employment, social agencies, or 
community programs 

d. ) Recruitment by co-sponsoring 

groups 

e. ) Program staff member assigned , — > — 

to recruitment 

f. ) Organized recruitment by 

current students/trainees 

g. ) Recommendations by current 

students/trainees 

h. ) Local residents used as 

recruitment aides 

i. ) Other (specify: ) 
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10. How many hours of volunteer time were donated to your program over the last 
program year? (If none, enter zero.) 


Individual Tutoring: 


hours 


Classroom Instruction: 


hours 


Classroom/Instructional Aide: 


hours 


Support Services: 


hours 


Recruitment: 


hours 



1 1. Where would you place your program on the following continua: 
(Mark an "x" on the appropriate line.) 



a.) 


Highly 






Individualized 


Prestructured 




Design 


Fixed Design 


b.) 


Emphasis on 


Emphasis on 




Academic 


Workplace or 




Skills _ 


Life Skills 


c.) 


Student 


Program 




Designed or 


Designed or 




Selected 


Selected 




Materials 


Materials 


d.) 


Reliance on 


Reliance on 




Nationally 


Criterion 




Normed tests 


Referenced Tests 
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12 . 



How would you characterize the Involvement of each of the following types of public 
and private organizations (other than your agency) with your program? For each 
row, check the appropriate column(s). 

Not Involved Involved In Provide Provided Other 

significantly in recruiting/ funding staff. (Specify 

involved planning referrals facilities. below* 

& other 
resources 



a. ) Local school 

district 

b. ) Community college 

c. ) Area voc-tech 

schools 

d. ) State and local 

employment and 
training agencies 

e. ) Literacy councils/ 

organizations 

f. ) Religious groups 

g. ) businesses or 

labor unions 

h. ) Representatives of 

special adult 
populations 

i. ) Other fraternal. 

voluntary or 

community 

organizations 

J.) Public libraries 

k. ) Hospitals 

l. ) Other state and 

local agencies 



47 

o 

ERLC 



A-15 



13. For your Instructional program, what Is the number of Instructors and volunteers 
who are . . . ? 

a. ) Full-time paid instructors: 

Part-time paid Instructors: ^ 

Volunteer Instructors: 

b. ) Teach In the program: Full-time Staff Part-time Staff Volunteers 

10-12 months/yr 

6-9 months/yr 

less than 6 months/yr 

c. ) Teach: Full-time Staff Part-time Staff Volunteers 

day classes only 

night classes only 

day and night 

d. ) Have taught classes in your Full-time Staff Part-time Staff 

.program: Volunteers 

less than 1 year 

1-3 years 

over 3 years 

e. ) Have completed: Full-time Staff Part-time Staff Volunteers 

some college or post- 

secondary education 

BA/BS 

MA or higher degree 

f. ) Are: Full-time Staff Part-time Staff Volunteers 

certified specifically in adult 

education 

certified in area other than 

adult education 

not certified _____ 

g. ) Besides teaching: Full-time Staff Part-tim^ Staff Volunteers 

have no other program 

responsibilities 

have other program 

responsibilities _____ 
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14. What percentage of your instructors have been provided with Initial orientation or 
in-service training within the past program year? 

% 

15. During the past program year, which of the following forms of in-service training 
have been provided to your instructional staff? (Check all that apply.) 

a. ) No in-service training provided (Skip to 15) 

b. ) Assignment to work in the 

classroom of a more experienced 
teacher or staff member 

c. ) Participation in curriculum 

development 

d. ) Coaching by supervisors or 

others 

e. ) Workshops and conferences 

f. ) Participation in related 

university courses 

g. ) Other (specify) 



49 

o 

ERIC 



A-17 



16. 


During the past program year, which of the following have been a malor thrust of 
the in-service training for vour instructors? fcheck all which apply.) 


a.) 


Improving reading 
instruction 




b.) 


Improving writing 
Instruction 




c.) 


Improving math instruction 




d.) 


Improving vocational 






instruction 




e.) 


Assessing 

students/trainees needs 




f.) 


Recruiting 

students/trainees 




g-) 


Instructing 

students/trainees with 
physical handicaps and 
learning disabilities 


— 


h.) 


Counseling and otherwise 
dealing with 
students’/trainees’ 
personal problems 


— 


i.) 


Providing instructors with 
occupational knowledge 
relevant to their students 


— 


j.) 


Relating instruction to 
students’/trainees* 
ethnic/cultural 
backgrounds 


— 


k.) 


Preventing student/trainee 
dropouts 




1.) 


Involving students/trainees 
in planning and evaluating 
their own programs of 
instruction 
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17. 


What percentage of teachers use each of the following? 
(Check appropriate responses.) 

Few 

None (less than 25%) 


Some 

(25% -50%) 


Most 

(over 50%) 


a.) 


Learning contracts 








b.) 


Student projects 








c.) 


Role play, learning games, 
simulations programs 








d.) 


Student participation in 
planning own program 








e.) 


Problem solving through 
discussion groups 








f.) 


Student participation in 
evaluating own learning 
gains 








g-) 


Student Journals and/or 

exchange letters with 

students 








h.) 


Use of existing students in 

new student orientation 








i.) 


Student/teacher prepared 
instructional materials 








J-) 


Modes of assessment of 

learning gains other than 

tests (e.g. portfolios) 
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18. To what extent does your program use each of the following learning 

environments? (In responding, assume that "veiy little" means less than 10% of 
total client Instructional time; "some" means 10% to 50%; and "a great deal" means 
over 50% of total client Instructional time. Check the appropriate column for each 
learning environment.) 

Very A Great 

Not at All Little Some Deal 

a. ) Individual Instruction, 

(e.g., one-on-one 

tutoring) 

b. ) Individual self-study 

with no Instructor or 

tutor present 

c. ) Small group Instruction 

for students with 

similar problems 

d. ) Classroom style 

Instruction with 1 or 

more aides 

e. ) Multi-media learning 

labs or centers 

f. ) Computer-assisted 

instruction 

g. ) Real or simulated 

workplace settings 

h. ) Other (specify: ) 



19. What is your program’s total budget, not including non-cash contributions for the 
year ending June 30, 199_? 



$ 
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20. Over and above the budget amount you provided In response to Question 19, do 
you receive any non-cash ln-kind and/or donated contributions? 

Yes No 

Roughly, how much of an increase would you need in your budget if you had to 
pay for the facilities, goods and services your program now receives as in-kind 
contributions? 

None, no in-kind received 

about a 25% increase 

about a 50% increase 

about a 75% increase 

about a 100% increase 

more than 100% 

(if over 100%, what’s your best guess? %) 

21. Approximately what percentage of your total budget (Question 19) is devoted to the 
following: (Answer using dollars or percentage, which ever is easier.) 



a.) 


Administrative Staff 


% 


$ 


b.) 


Instructional Staff 


% 


$ 


c.) 


Counseling Staff 


% 


$ 


d.) 


Clerical and other staff 


% 


$_ 


e.) 


Instructional 
materials/ equipment 


% 


$ 


f.) 


Facilities 


% 


$ 


g) 


Utilities 


% 


$ 


h.) 


Custodial Services 


% 


$ 


i.) 


Office 

equipment/fumiture/supplles 


% 


$ 


j) 


Other (specify): 


% 


$ 


k.) 


Other (specify): 


% 


$ 




TOTAL 


100 % 


$ * 



* Should be equal to amount in Item 19 



22. Approximately what percentage of your total budget (see Item 19) comes from each 
of the following sources? (Answer using dollars or percentage, whichever is easier.) 



a. ) Federal government (all sources) % $ 

b. ) State government (all sources) % $ 

c. ) Local government (all sources) % $ 

d. ) Private Sector Donations % $ 

e. ) Other % $ 

TOTAL 100 % $ * 



• Should be equal to amount in Item 19. 
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Appendix A3 

Student Instructional Status and 
Attendance Form 




1. Current Program Status: 

□ Active: Received one or more hours of Instruction (Go to 3) 

□ Inactive: Received no instruction (Go to 2) and stop 

2. Reason for Not Receiving Instruction 

D Student no longer attends, reason unknown 

□ Completed instructional program, not interested in going further 

□ Completed highest level of instruction offered 

□ Completed requirements of employer/ other agency /other program 

□ Forced to leave by personal circumstances— moved, changed Job, health, family 
obligations, etc. 

□ Did not complete program, but left expressing satisfaction 

□ Did not complete program and left expressing dissatisfaction 

□ Transferred to another site/program 

□ Participation ended for other reasons 



3. 


Learning Environment(s) 
(Mark All That Apply) 


4. 


Support Services Used 
(Mark All That Apply) 


□ 


Class with teacher only 


□ 


Child care 


□ 


Class with teacher and aide 


□ 


Transportation 


□ 


Computer assisted lab 


□ 


Health Services 


□ 


Vocational lab 


□ 


Counseling 


□ 


Learning Lab 


□ 


Job Search assistance 


□ 


Individual tutor 


□ 


Translator services 


□ 


Self-study, no instructor 


□ 


Financial assistance 


□ 


Other 






5. 


Approximate Class Size 
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Days and Hours Scheduled or Attendance Pattern for Typical Student 

Days Scheduled Hours Scheduled 

Per Week: Per Week: 



Monthly Student Attendance Record 



Month: 




Month: 




Month: 




Total Days 
Attended 


Total Hours 
Attended 


Tefal 

Days 

Attended 


Total Hovers 
Attended 


Total 

ftays 

Attended 


Total 

Hours 

Attended 


Month: 




Month: 




Month: 




Total Days 
Attended 


Total Hours 
Attended 


Total 

Days 

Attended 


Total Hours 
Attended 


Total 

Days 

Attended 


Total 

Hours 

Attended 
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Appendix A4 



Student Follow-up Telephone Suhvey 
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1. I understand that you attended the program at about six months ago. Is 

that correct? 

O Yes □ No (If the respondent anran no to this question, the Interview i. terminated.! 

2. Did you learn what you wanted to learn from the instruction that you received? 

□Yes □ No 

3. To what extent did the educational class or training program help you to improve 
reading/writing skills? 

□Yes □ Somewhat CD Not At All 

4. To what extent did the educational class or training program help you to improve 
math skills? 

□Yes CD Somewhat CD Not At All 

5. To what extent did the educational class or training program help you to improve 
speaking and listening skills? 

□Yes □ Somewhat CD Not At All 

6. To what extent did the educational class or training program help you to enter a 
vocational or job training program? 

□Yes □ Somewhat □ Not At All 

7. To what extent did the educational class or training program help you to get a job or 
a better job or qualify for higher pay? 

□Yes □ Somewhat CD Not At All 

8. To what extent did the educational class or training program help you to improve Job 
performance? 

□Yes □ Somewhat □ Not At All 

9. To what extent did the educational class or training program help you to qualify for 

United States citizenship? 

□Yes □ Somewhat □ Not At All 

10. To what extent did the educational class or training program help you to feel better 
about yourself? 

□Yes □ Somewhat CD Not At All 

1 1 . To what extent did the educational class or training program help you to contribute 
better to your family and community? 

□Yes □ Somewhat □ Not At All 
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12. To what extent did the educational class or training program help you to help your 
children with schoolwork? 

□Yes □ Somewhat □ Not At All 

13. To what extent did the educational class or training program help you to become less 
dependent on others for help? 

□Yes □ Somewhat □ Not At All 

14. To what extent did the educational class or training program help you to mair» others 
feel better about you? 

□Yes □ Somewhat □ Not At All 

15. To what extent did the educational class or training program help you to continue 
your education? 

□Yes □ Somewhat □ Not At All 

16. Did you attend the educational class or training program to get a GED? 

□Yes □ No 

17. Did you attend the educational class to earn a regular High School Diploma? 

□Yes □ No 

18. What was your major reason for leaving your class or Instructional program? 

□ Left Satisfied. . .completed program 

□ Left Satisfied. . .completed required attendance 

□ Left Satisfied. . .got what went for/achieved personal goals 

□ Left Satisfied. . .other (e.g., enrolled In school, got job) 

□Left/Outside Events. . .personal Illness, health problems 

□ Left/ Outside Events. . .family responsibilities (Including child care) 

□Left/Outside Events. . .transportation problems 

□ Left/Outside Events. . .change of Job or work responsibilities 

□ Left/Outside Events. . .other reasons not to do with the course Itself 

□ Left Due To Courses. . .too many forms/to many tests to take 
□Left Due To Courses. . .personal embarrassment/personal discomfort 

□ Left Due To Courses. . .lack of progress/dissatisfied with program 
□Left Due To Courses. . .took too much time & energy (e.g., too far to drive) 

□ Left Due To Courses. . .Interfered with other (preferred) activities 

□ Left Due To Courses. . .worried about personal safety (e.g., bad neighborhoods) 

□ Left Due To Courses. .. .other reasons reflective of the program 
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Questions 19 - 25 are asked of those clients who had a job 
when they enrolled. 



19. I see that you were working when you enrolled in the program. 

Are you working now for pay? 

□ Yes (go to 20) O No (go to 22) 

20. Do you still have that same job you had when you enrolled in the program? 

□ Yes (go to 21) □ No (go to 23) 

21. Did what you learned in the program help you with that job? (i.e. the same job you 
had when you enrolled) 

□ Yes (go to 30) □ No (go to 30) 

22. Have you worked regularly at any time since you left the program? (asked of those 
not working now.) 

□Yes (go to 25) 



□ No (go to 30) 



23. Did what you learned help you get a better job than you had? (asked of those who 
changed jobs.) 

□Yes l*o to 24) □ No (go to 24) 

24. Did what you learned in the program help you in your cun-ent Job? (asked of those 
who changed jobs.) 

□ Yes (go to 30) □ No (go to 30) 

25. Did what you learned in the program help you with that job? (asked of those not 
currently working, but who worked regularly since leaving the program.) 

□ Yes (go to 30) □ No (go to 30) 



Questions 26 - 29 are asked of those clients who were not 
working when they enrolled. 



26. I see that you were not working when you enrolled in the program. Are you working 
now for pay? 

□ Yes (go to 28) □ No (go to 27) 

27. Have you worked at any time since you left the program? 

□ Yes (go to 28) □ No (go to 30) 

28. Did what you learned in the program help you get the Job? 

□ Yes (go to 29) □ No (go to 29) 

29. Did what you learned in the program help you with that job? 

□ Yes (go to 30) □ No (go to 30) 

30. Do you have children under the age of 6? 

□Yes (go to 3i) □ No (go to 32) 

3 1 . How often do you read to or with your children? 

□Almost Never 

□About Once Or Twice A Month 
□About Once A Week 
□Nearly Every Day 



32. Do you feel you need or would like additional educational classes or training? 

□ Yes (go to 33) □ No (go to 34) 
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33. What Is your reason for wanting additional classes or training? 

□Academic. . . reading/writing 

□Academic. . . mathematics 
□Academic. . . understanding English 

□ Credentials . . .to obtain GED/hlgh school diploma 

□ Credentials . . .enter vocational or job training program 

□ Credentials . . .enter college 
□Employment. . .learn a trade/ skill 
□Employment. . .get a Job or better Job 
□Employment. . .improve Job performance 
□Employment. . .increase Income 

□Employment. . .satisfy employer or program requirements 

□Personal. . . prepare for citizenship 

□Personal. . . improve self/feel better about myself 

□Personal. . . contribute to children 

□Personal. . . contribute to church, community 

□Personal. . . become less reliant on others 

□other Reason 

For the next few questions, describe how you usually felt about the 
instruction you received in your vocational/adult education program. 

34. Were the lessons at a level you could understand... 

□Always □ Most Of The Time 

□Sometimes □ Never 

35. Were the instructional materials helpful... 

□Always □ Most Of The Time 

□ Sometimes □ Never 

36. Did your teacher work with small groups of students... 

□Always □ Most Of The Time 

□ Sometimes □ Never 

37. Was the teacher or aide available to help you individually with the lessons... 

□Always □ Most Of The Time 

□ Sometimes □ Never 
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38. Was there enough practice time In the class... 

□Always □ Most Of The Time 

□ Sometimes □ Never 

39. Were questions from students encouraged by the teacher... 

□Always □ Most Of The Time 

□ Sometimes □ Never 



40. Were your questions answered... 

□Always □ Most Of The Time 

□ Sometimes □ Never 

41. Did you have access to computers In your instructional program? 

□Yes □ No 

42. Were they helpful to you... 

□Always □ Most Of The Time 

□ Sometimes □ Never 

43. Are you attending any educational classes or training program now? 

□ Yes (go to 44) □ No (go to 45) 

44. What kind of educational class or program are you attending? 

□English Language Skills 

□ Math Skills 

□ GED/High School 
□Vocational/ Job Training 

□ Community College/College Level 

□ Citizenship 

□ other 

45. Do you plan to enroll in any educational classes or training program In the future? 

□ Yes (go to 46) □ No (end Interview) 
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46. When do you plan to take the classes/program? 

□ Before Next Year 
□Next Year 

□Within The Next Three Years 
□Longer Than Three Years From Now 

47. What educational classes or training program do you plan to take? 
□English Language Skills 

□Math Skills 

□ GED/High School 
□Vocational /Job Training 

□ Community College/College Level 

□ Citizenship 

□ Other 
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Appendix A5 



Initial Job Placement Information Form 



Name: ■ 

Vocational Training Program: 
Program Completion Date: 



A. Placed In Job at Program Completion or Exit? 
□ Yes □ No 



If no, Is Individual seeking employment? 

□ Yes □ No 

If yes: If no, why not? 

1. Name of Company: 

Address: 

Telephone: 

Supervisor’s Name: 

2. Nature of Job: 



3. Is Job Related to Training? 
□ Yes □ No 



4. 

5. 

6 . 

7. 

8 . 



Date Hired: 

Date Job Begins: 

Number of Hours to be Worked Per Week: 

Starting Gross Wage Per Hour: 

Does Job Include Medical Benefits? 

□ Yes □ No 



B. Enrolled for Further Education 
□ Yes □ No 
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Appendix A6 

job Placement Follow-up Questionnaire 




1 . 



Are you currently employed? 



□ Yes 

(Continue Survey) 

2. Who do you work for? 

3. Have you been employed continuously since completion of or exit from the vocational 

training program? □ Yes 

□ No 

4. Please describe your current job. 



5. Is your current job: 

Same as when originally hired □ 

A promotion from original job □ 

A demotion from original job □ 

Different job, but neither a promotion nor a demotion □ 

6. What is your current gross hourly wage? $ per hour 



□ No 

If no, have you been employed at any time since 
completion of or exit from program? 

□ Yes 

□ No 



7 . Are you being paid more, less, or the same as the first Job you obtained after you 
completed or exited from the vocational training program? 

More □ 

Less □ 

The Same □ 

8. How many hours do you work per week? 



9. Do you receive medical benefits from your employer? 

□ Yes 

□ No 

10. Are there opportunities for you to advance at the place you work? 

Many Opportunities G 
Few Opportunities □ 

No Opportunities □ 

Explain: 



1 1 . How satisfied are you with the vocational training you received from the program? 



Very Satisfied □ 

Satisfied □ 

Dissatisfied G 

Very Dissatisfied G 



12. How satisfied are you with the basic skills instruction you received from the 
program? 



Very Satisfied 


G 


Satisfied 


G 


Dissatisfied 


G 


Very Dissatisfied 


G 
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13. Would you recommend the program to others? 

□ Yes 

□ No 

Explain: 



14. Are you currently enrolled In any school, educational, or vocational training program? 

□ Yes 

□ No 

Specify: 



15. How satisfied are you with your present Job? 



Like it very much □ 

Like It somewhat □ 

Neither like It nor dislike it □ 

Dislike It somewhat □ 

Dislike it a lot □ 

Why? 



16. What changes in the program do you suggest for the future? 



Appendix A7 

Employer Follow-up Questionnaire 



Name of Employee: 
Company Name: 
Type of Business: 



1 . Is currently employed by you or your company? 

□ Yes (Continue with Survey.) 

D No [If no. ask why not and stop.) 



2. Briefly describe this employee’s current job. 



3. How many hours per week does individual work: 

4. Is his/her current Job: 

Same as when originally hired E 

Promotion from original Job with company □ 

Demotion from original job with company D 

Different job, but neither a promotion nor demotion □ 

5. What is employee’s present gross hourly wage? $ per horn-. 



6. Is this employee’s wage higher, lower, or the same as when originally hired? 

Higher □ 

Lower O 

The Same D 

7. Does employee receive medical benefits? 

□ Yes □ No 

8. Are there opportunities for this employee to advance at your company? 

Many Opportunities D 

Few Opportunities . D 

No Opportunities D 

Explain: 
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9 . 



Compared to other Individuals at the same Job level, how would you rate this 
employee’s lob skills? 

Above Average □ 

Average □ 

Below Average □ 



10 . 



Is this employee’s basic language and math skills sufficient 
current Job? 



to perform his/her 



Yes No 

Language skills are a problem D D 

Math skills are a problem Q HI 



11.. Does this employee have the skills necessary to advance at your company? 

□ Yes 

□ No 

Explain: 



12. In general, how satisfied are you with this employee's work? 

Very Satisfied □ 

Satisfied □ 

Somewhat Satisfied □ 

Dissatisfied □ 
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Appendix B 

Measurement of English Language 
Proficiency of Limited English 
Proficient Adults 



For programs which target students or trainees who are not native speakers of E nglish 
and are limited-English proficient, the measurement of English language proficiency is 
important for two reasons. First, it will provide the data for describing how well individual 
participants are progressing in increasing their English language skills. Second, it will 
provide the basis for assessing how well the program has been able to increase Job-related 
English language proficiency. 

In order to select an appropriate language proficiency instrument, it is important to 
consider specific project goals and needs, as well as the characteristics of the instrument 
itself. Thus, the adequacy of any language proficiency measure should be assessed 
according to the criteria below: 

► Is the instrument appropriate for adults ? 

► Does the measure assess the appropriate language areas? 

► Is the measurement appropriate, i.e., does it take a discrete-point or 
integrative approach? 

► Is the instrument technically sound, i.e., is the validity and reliability of the 
instrument adequate? 

The first criterion requires project staff to assess the appropriateness of the instrument for 
limited English proficient (LEP) a dult s. This step is extremely important since there may 
be the temptation to simply adopt an existing instrument designed for LEP school-age 
youth. In rare instan es, such an Instrument could be used. However, in the majority of 
cases, such use would be Inappropriate and the resulting information would be 
Inadequate. If an Instrument is not designed for an adult population, it will be critical to 
evaluate the content of the measures to make certain that it is appropriate for adults. 

The second criterion requires that an instrument be selected which measures the 
language skill areas which need to be assessed. The areas that can potentially be 
measured Include all components of listening, reading, speaking and writing. Before an 
instrument is selected, the project staff need to decide on the specific skill areas to be 
measured. This Judgment should be made on the basis of project goals and vocational 
skill area(s) in which training is provided. In projects which train in vocational skill areas 
that require basic reading and writing (clerical, general office skills, and word processing), 
assessment of literacy 1 is necessary. In projects which focus on instructional areas that 



‘Literacy refers to reading and writing skills. 
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do not necessarily require literacy (cosmetology and building trades), oracy skills 2 need to 
be measured. 

The third criterion relates to the measurement approach taken by a particular language 
proficiency Instrument. The two possible approaches are discrete-point or Integrative. If 
the focus is on the assessment of individual aspects of language such as gra mmar , 
vocabulary, etc. , the measurement approach Is considered to be discrete-point. However, 
if the orientation is toward assessing an individual’s functional language ability, the 
approach is integrative. At the simplest level, this latter approach includes the 
assessment of an individual's capability to listen to a basic conversation and to provide 
appropriate responses to questions regarding name, home address, listing of family 
members, etc. At a more sophisticated level, the assessment could require that an 
individual participate in a simulated job interview or dialogue appropriate to a work 
setting. 

Since a general goal at the local level is to assist participants to communicate in En glis h 
in a job-related setting, a language proficiency instrument that takes an integrative 
approach is recommended. However, it should be noted that a discrete-point type 
instrument may also be appropriately used at the project level once an integrative 
assessment has been made and specific skills areas that need to be assessed for 
diagnostic purposes have been identified. 

The fourth criterion concerns the technical qualities of the instrument. For example, it 
will be important to determine whether the testing materials Include standardized 
administration and scoring procedures. Whether it is a published or project developed 
instrument, it will be important to review validity and reliability data in order to judge its 
adequacy. 

The above four criteria provide a structure for project staff to evaluate and select language 
proficiency assessment instruments. It is important that any instrument be carefully 
evaluated in order to ensure the selection of one which is technically sound, and 
appropriate for the project and the language proficient levels of its students. 
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a Oraey relates to skills related to listening and speaking. 
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13. What changes. If any. should the program make in its training program to enable its 
trainees to be more valuable employees to your company? Do individuals need more 
training? If so. In what areas? 



